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Background / Context: 

Description of prior research and its intellectual context. 

Over the last deeade, New York City has been the site of a systemwide high school reform 
effort that is unprecedented in its scope and pace. Since 2002, the school district has closed more 
than 20 failing high schools, opened more than 200 new secondary schools, and implemented a 
centralized high school admission process in which approximately 80,000 students a year 
indicate their school preferences from a wide-ranging choice of programs. 

At the heart of these reforms lie the new schools that in this report are called “small 
schools of choice” (SSCs) — small, academically nonselective, public high schools that were 
opened between 2002 and 2008. Serving approximately 100 students per grade in grades 9 
through 12 and open to students at all levels of academic achievement, the SSCs in this study 
were created to serve the district’s most disadvantaged and historically underserved students. 
Prior to the 2002-2003 school year, these students would have had little option but to enroll in 
one of the city’s large, zoned high schools when they made the transition from eighth to ninth 
grade. Many of the large schools were low-performing, with graduation rates below 50 percent. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

This report presents encouraging findings from an unusually large and rigorous study, 
supported by the Bill & Melinda Gates Foundation, of the effects of SSCs on students’ academ- 
ic achievement in high school. It emerges at a moment when policymakers, practitioners, and 
researchers have identified the high school years as the point of greatest weakness within the 
education pipeline. The rationale for this collective focus is clear: far too many students drop out 
of high school, and the consequences of entering adult life without a high school diploma are 
increasingly grave. 

Setting: 

The New York City public school system is the largest in the United States, with over 1.1 
million students enrolled in more than 1,600 schools. Over the past decade, it has been the site of 
an ambitious effort to reform the high school system, of which the creation of SSCs was a central 
part. Beginning in 2002, the New York City Department of Education (DOE) accelerated and 
expanded efforts that had been under way since the mid-1990s to close large, low- performing 
schools and open new small schools in their stead. These reform efforts were supported by a 
consortium of funders led by the Bill & Melinda Gates Eoundation — which ultimately invested 
over $150 million in New York City — and were implemented in partnership with the teachers 
and principals unions. The resulting changes in the high school landscape transpired with 
unprecedented scale and rapidity. By 2008, 23 high schools with graduation rates below 45 
percent had been targeted for closure, and 216 new small schools, of which 123 were SSCs, had 
been opened. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features or characteristics. 

The full sample for the present study represents 297 SSC lotteries with 12,978 observations 
on lottery winners and 17,981 observations on control group members who were eighth-grade 
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HSAPS participants in the 2004-2005 through 2007-2008 school years. Because some students 
partieipate in more than one lottery, the total number oi students in the full sample (21,085) is 
less than the total number of student observations (30,959). 

Table 1 provides strong evidenee of the baseline equivalenee of the present sample of 
SSC lottery winners and eontrol group members. This table compares the two groups in terms of 
a number of eharaeteristies that typieally prediet high sehool sueeess. The first eolumn in the 
table reports means for SSC lottery winners. The seeond eolumn reports means for eontrol group 
members. The third eolumn reports differenees between means for the two groups and the fourth 
eolumn reports the statistieal signifieanee (probability value, or p-value) of these differenees or 
the probability that an observed differenee for a sample eould have oeeurred by ehanee if there 
were no dijference in the population. As ean be seen, all differenees are quite small and none is 
statistically significant. Thus, differences in mean future outeomes for SSC lottery winners and 
their eontrol group eounterparts provide valid estimates of the effects of winning an SSC lottery. 

On average, both groups (SSC lottery winners and the eontrol group members) are 
students of eolor (roughly 90 pereent are either Hispanie or blaek) and qualify for free-/redueed- 
prieed luneh. Roughly 17 pereent of the sample is overage for grade, about 70 percent were in 
the bottom two levels of reading profieieney (aecording to the NY State eighth grade reading 
test), and about 65 pereent were in the bottom two levels of math profieieney (aeeording to the 
NY state eighth grade math test). 

Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 

While the distriet established a variety of small sehool models, ranging from transfer sehools 
designed to serve students who had struggled in eonventional high sehools to speeialized sehools 
intended to serve the distriet’ s highest-performing students, the predominant model was the 
small sehool of ehoiee, whieh, notably among the other sehool types, was academieally 
nonseleetive and small not only in size but also in funetion. That is, struetures sueh as redueed 
teaeher load and eommon planning time (in whieh teaehers meet together to diseuss their 
students’ progress and problems) were reeommended to ensure that all students were known well 
and to promote strong, sustained relationships between students and teaehers. SSCs also had four 
other essential features: 

• SSCs were predominantly loeated in disadvantaged eommunities whose neighborhood high 

sehools were closing. 

• SSCs were established via a demanding and competitive proposal proeess that emphasized 

the eommon design prineiples of aeademie rigor, personalization, and eommunity 
partnerships. This proeess required a prospeetive sehool leadership team to artieulate an 
educational philosophy and demonstrate how it would motivate teaehers, community 
members, and partner organizations around it. Additionally, the new sehool leadership 
had to develop a viable improvement strategy from the ground up. 

• SSCs benefited from an infusion of outside resourees: new prineipals and teaehers, 

partnerships with intermediary organizations that had expertise in starting new schools, 
and start-up funding from the district and its philanthropic partners. 

• SSCs reeeived policy protections during their start-up period, ineluding opening with only 

one founding grade of students (ninth grade) and having aeeess to supports to faeilitate 
procurement and hiring — sueh as speeial training for school principals and teachers; an 
amendment to the eolleetive bargaining agreement, whieh gave prineipals more hiring 
diseretion; and the eonversion from a management system of regional offiees to one in 
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which schools had greater eontrol over their budgets and edueational programs. 

Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 

In the spring of 2004, the eity introdueed the High Sehool Applieation Proeessing System 
(HSAPS), a eentralized ehoiee proeess that was to govern the plaeement of all entering ninth- 
grade students. HSAPS uses an objeetive, eomputer-based proeess to assign about 72,500 
entering ninth-graders annually to about 400 publie high sehools. When they are in the eighth 
grade, students who partieipate in HSAPS indieate, in order of preferenee, up to 12 high sehools 
they would like to attend. Eaeh year, some sehools have more applieants than seats available. 
When this oeeurs at an SSC, a lottery is ereated within HSAPS that randomly determines whieh 
students are assigned to that sehool. 

The existenee of these lotteries provides an unpreeedented opportunity to launeh a 
rigorous study of the effeets of this group of sehools on student aeademie aehievement, beeause 
the lotteries ereate two randomized groups among students who ehose a given SSC — those who 
won its lottery and were assigned to the SSC and those who lost its lottery and were assigned 
elsewhere. Future outeomes for these two groups ean be eompared to obtain valid estimates of 
the effeets of SSCs on student aehievement. The lotteries ereated by HSAPS together with the 
unusually large size of the randomized sample they produeed allow for a high degree of validity 
and preeision in the present analyses. Thus, one ean have eonsiderable eonfidenee in them. Using 
these lotteries as the basis for its analysis, this report presents the estimated effeets of enrolling 
in a small sehool of ehoiee versus enrolling in one of the other high sehools that are available to 
the average ineoming ninth-grader.^ 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

The analysis presented in this report uses data from the high sehool admissions proeess to 
identify a sample of students who ehose SSCs, but who — beeause their ehosen SSC had more 
applieants than seats available — were assigned via lottery either to that sehool or to a 
subsequent ehoiee on their list. The analysis ineludes four annual eohorts of students who 
entered high sehool in the fall of 2005, 2006, 2007, and 2008, respeetively — a total of 21,085 
students who applied to the 105 SSCs that were oversubseribed, and for whieh lotteries were 
held, during the study period. 

Most of the sehools attended by students who did not enroll in an SSC were older and 
larger than the SSCs: all SSCs were ereated sinee 2002 while two-thirds of the sehools attended 
by the non-SSC enrollees were established before then, and the ninth-grade elasses averaged 129 
students in SSCs and 635 students in the non-SSC sehools.^ However, it is important to 
remember that the SSCs are not being eompared with the large, failing sehools they replaeed but 
rather with a wide range of sehools that were also operating in a reform-rieh atmosphere. 



2 

To estimate the effects of enrolling in an SSC, the estimated effects of winning an SSC lottery are adjusted to account 
for the proportion of SSC lottery winners who do not enroll in an SSC and the proportion of control group members who 
do enroll in an SSC, using a well- known statistical approach called instrumental variables analysis. 

3 

While the schools attended by non-SSC enrollees were significantly larger, some of those larger schools (for 
approximately one-eighth of those students) had structures such as small learning communities in place to increase the 
level of personalization. 
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Findings / Results: 

Description of the main findings with specific details. 

SSCs have a substantial positive impaet on the transition into high sehool during ninth grade, 
aeeording to data using all four eohorts (see Table ES.l): 

• SSC enrollees were 10.8 percentage points more likely than the students who enrolled in 

other schools to earn 10 or more credits during their first year — 73.1 percent compared 
with 62.3 percent. 

• SSC enrollees were 7.8 percentage points less likely to fail more than one core subject (39 

percent compared with 46.8 percent). 

• Combining these two indicators, 58.5 percent of SSC enrollees were on track to graduate in 

four years compared with 48.5 percent of their counterparts who attended a different type 
of school — a 10 percentage point difference. 

• During the first year of high school, SSC enrollees earn almost one full credit more (0.9 

credit) toward graduation than do their control group counterparts. 

These positive effects on the transition into high school during ninth grade were seen 
among nearly all subgroups as defined by students’ academic proficiency, socioeconomic status, 
race/ethnicity, and gender. 

For all students, second- and third-year follow-up data indicate that these effects are 
sustained and/or increased as they continue through high school (see Table ES.l). For the first 
cohort of students (the only cohort for whom there are four years of follow- up data), the 
evidence indicates that SSC improvements in students’ academic progress and school 
engagement during the early years of high school translate into higher rates of on-time 
graduation after four years: 

• SSCs increase overall graduation rates by 6.8 percentage points, from 61.9 percent for 

students who attend schools other than SSCs to 68.7 percent for SSC enrollees. 

• A majority of the SSC effect on graduation rates reflects an increase in receipt of New 

York State Regents diplomas. 4 For this type of diploma, students must pass a series of 
Regents examinations with a score of 65 points or above and pass all of their required 
courses. 

• SSCs increase the proportion of students (by 5.3 percentage points) who passed the English 

Regents with a score of 75 points or higher, the threshold for exempting incoming 
students at the City University of New York from remedial courses. They did not have an 
effect on math Regents exams. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

These findings speak to the nation’s current focus on high school reform. Much of the 
national discussion focuses on three areas where the education community has struggled to 
demonstrate success: (1) improving the academic outcomes of the most disadvantaged students, 
particularly with respect to high school graduation and college readiness; (2) identifying turn- 
around strategies for historically underperforming schools; and (3) implementing effective 



4 

Although the estimated effect of SSCs on the overall high school graduation rate is statistically significant, estimates of 
SSC effects on graduation rates by type of diploma (p = 0.07) miss the standard of statistical significance established for 
this study (p = 0.05). Thus, comparisons of effects across diploma types are suggestive only. 
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interventions at scale. This study sits at the nexus of all three themes, and its findings demon- 
strate that, in a relatively short period of time, an effective model can be implemented at scale 
and can improve the academic trajectories of large numbers of traditionally underserved 
students. 

The effects of small high schools of choice described in this report should be understood 
through three important lenses: their scale, the particular package of reforms they represent, and 
the group of highly disadvantaged students for whom they occurred. 

Effecting Change at Scale. At capacity, the 105 SSCs in the study sample will serve over 
45,000 students. That is roughly equivalent to the entire high school population of Houston, 
which is the seventh largest school district in the country. Readers should understand the 
magnitude of the present report’s findings in that context — imagine,ybr a school district the 
size of Houston, increasing the percentage of ninth-graders who are eligible for on-time promo- 
tion by 10.8 percentage points, the percentage of black males in ninth grade who are on track to 
graduate by 8.5 percentage points, or the percentage of high school graduates by 6.8 percentage 
points. Given the scale of the SSC initiative, even seemingly minor gains can be understood as 
affecting thousands of high school students. In fact, the 6.8 percentage point increase in four- 
year graduation rates is roughly equivalent in size to one-third of New York City’s gap in 
graduation rates between white students and students of color. Additionally, because the reported 
effects of SSCs are not the product of a small, targeted intervention but rather of a large system 
of small schools, the effects can be understood as reflecting the mean performance of a model 
implemented at scale. Reported effects are not the product of the best or most popular of the 
SSCs, but of 105 schools on average. In other words, the findings represent a real-world test of 
an intervention launched at the scale of a large-sized urban school district. 

The SSC Package of Reforms, Students enrolled in SSCs did not just attend schools that 
were small. SSC enrollees attended schools that were purposefully organized around smaller, 
personalized units of adults and students, where students had a better chance of being known and 
noticed, and where teachers knew enough about their charges to provide appropriate academic 
and socioemotional supports. SSCs were not only new but were mission-driven. Their recent 
establishment via a demanding authorization process, which rejected more school proposals than 
it approved, required that a prospective school leadership team articulate an educational 
philosophy and demonstrate how it would motivate teachers, community members, and partner 
organizations around it. And the district’s commitment to acting as a steward for new schools 
throughout the start-up period generated a set of supports and protections as these schools got up 
and running. Finally, SSCs benefited from an influx of external ideas, talent, and resources. 

Serving Disadvantaged Students, SSCs were intended to be a viable and accessible 
option for the district’s most disadvantaged students, and over the course of the study period, 
they served a population that almost exclusively comprised low-income students of color. The 
fact that SSCs targeted and served this population gives the reported findings even greater policy 
significance, as it is precisely economically disadvantaged students of color who find themselves 
at the bottom end of the nation’s persistent achievement gap, and who are least likely to graduate 
from high school on time, if at all. Furthermore, the robust positive SSC effects for many 
different types of students, including young men of color, hold out great hope for educational 
policymakers, practitioners, and researchers who wish to effect change, by demonstrating that it 
is possible to transform a large number of high schools in ways that benefit many disadvantaged 
students. 
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Appendix A, References 
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Appendix B, Tables and Figures 

Not included in page count. 



New York City Small Schools of Choice 

Table 1 



Baseline Cbaracteristics of SSC Lottery Participants: 
First Year of High School, Cohorts 1 to 4 



Characteristic (%) 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P-Value for 
Estimated 
Difference 


Race/ethnicity 


Hispanic 


47.3 


47.9 


-0.6 


0.480 


Black 


43.6 


43.2 


0.4 


0.641 


Other 


7.9 


7.5 


0.3 


0.428 


Male 


46.0 


45.5 


0.5 


0.525 


Eligible for free/reduced-price lunch 


84.0 


84.5 


-0.5 


0.467 


English language learner 


8.4 


7.6 


0.8 


0.114 


Special education** 


6.6 


6.7 


-0.1 


0.826 


Overage for 8th grade'’ 


16.7 


18.1 


-1.4 


0.153 


8th-grade reading proficiency** 


Did not meet standards (level 1) 


6.9 


6.6 


0.3 


0.486 


Partially met standards (level 2) 


60.5 


61.4 


-0.8 


0.328 


Fully met standards (level 3) 


28.4 


27.6 


0.8 


0.287 


Met standards with distinction (level 4) 


0.7 


0.7 


0.1 


0.580 


8th-grade math proficiency** 


Did not meet standards (level 1) 


18.8 


19.2 


-0.3 


0.628 


Partially met standards (level 2) 


45.1 


44.9 


0.3 


0.759 


Fully met standards (level 3) 


32.8 


31.9 


0.9 


0.238 


Met standards with distinction (level 4) 


2.3 


2.2 


0.1 


0.598 


Total number of student observations = 30,959 











SOURCES: MDRC's calculations use High School Application Processing System and New 
York City Department of Edueation (DOE) state test data for eighth-graders from the 2004-2005 
to 2007-2008 sehool years, as well as data from DOE enrollment files from the 2005-2006 to 
2008-2009 sehool years. 

NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for 
the differenee between SSC lottery winners and eontrol group members are obtained from a 
regression of a given baseline eharacteristic on a series of indieator variables that identify 
each lottery plus an indicator variable that equals 1 for lottery winners and 0 for lottery losers. 
The coefficient on the latter indicator variable equals the differenee in the mean baseline 
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characteristic for lottery winners and control group members. The value for control group 
members equals the corresponding value for SSC lottery winners minus the estimated difference 
between lottery winners and control group members. To facilitate eomputation, all variables are 
centered on the mean value for the lottery they represent. This approaeh is equivalent to directly 
accounting for each lottery by adding a 0/1 indieator variable for it (Wooldridge, 2000). In some 
eases, rounding may cause slight diserepancies. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members refieeted by the full set of baseline characteristics in 
the table. The resulting chi-square value is not statistieally signifieant (p-value = 0.387). 

Cohorts 1, 2, 3, and 4 eonsist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respectively. 

“This sample includes special education students who can be taught in the regular classroom 
setting. Speeial education students classified by the DOE as requiring collaborative team 
teaching services or self-contained elasses are not part of the sample. 

'’Lottery partieipants are elassified as "overage for eighth grade" if they were 14 or older on 
September 1 of the eighth-grade school year. 

‘’Students seoring at proficiency levels 1 and 2 are not considered to be performing at grade level 
for state math and reading exams. Due to missing test seores, the sum of levels 1-4 may not add 
to 100 percent. 
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New York City Small Schools of Choice 



Table 2 

Estimated Effects of SSC Enrollment in Years 1 to 4 of High School 













P-Value 




Target 


Control 




Effect Size 


for 




SSC 


Group 


Estimated 


(Standard 


Estimated 


Outcome 


Enrollees 


Counterparts 


Effect 


Deviation) 


Effect 



Year 1 of high school (cohorts 1 to 4) 



9th-grade on-track indicator® (%) 58.5 

Earned 10 or more credits 73.1 

Failed more than 1 semester of a 
core subject 39.0 

Total credits earned toward graduation'’ 11.3 



48.5 


10.0 


** 




0.000 


62.3 


10.8 


** 




0.000 






** 






46.8 


-7.8 






0.000 


10.4 


0.9 


** 


0.21 ** 


0.000 



Total number of student observations = 29,81 1 



Year 2 of high school (cohorts 1 to 3) 



Famed 20 or more credits (%) 69.4 

Total credits earned toward graduation'’ 22.3 

Total number of student observations = 21,822 



58.3 11.1 ** 0.000 

19.8 2.6 ** 0.31 ** 0.000 



Year 3 of high school (cohorts 1 and 

2i 

Earned 30 or more credits (%) 69.5 62.4 7.1 ** 0.000 

Total credits earned toward graduation'’ 32.2 29.7 2.4 ** 0.23 ** 0.000 

Total number of student observations = 13,297 



Year 4 of high school (cohort 1) 



Graduated from high school 


68.7 


61.9 


6.8 * 


0.013 


Local diploma granted 


24.6 


21.9 


2.8 


0.261 


Regents diploma granted 


39.5 


34.6 


4.9 


0.074 


Advanced Regents diploma granted 


4.4 


5.5 


-1.1 


0.366 



Total number of student observations = 5,363 

SOURCES: MDRC's calculations use High School Application Processing System data from 
eighth-graders in 2004-2005 to 2007-2008, as well as data from New York City Department of 
Edueation attendanee, eourse eredits, Regents exam, transaetional, and enrollment files from the 
2005-2006 to 2008-2009 sehool years. 
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NOTES; This table presents the estimated effeets for students who have follow-up eourse eredits 
data. Appendix A deseribes how values in the eolumn labeled "Target SSC Enrollees" are 
estimated. Appendix A also deseribes how values in the eolumn labeled "Estimated Effeet" are 
estimated. Values in the eolumn labeled "Control Group Counterparts" are differenees between 
eorresponding values in the first and third eolumns. 

A two-tailed t-test was applied to the estimated effeet. Statistieal signifieanee levels are 
indieated as: ** = 1 pereent; * = 5 pereent. 

Cohorts 1, 2, 3, and 4 eonsist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respeetively. 

^The on-traek eomposite measure indieates whether students earned at least 10 eredits in their 
first year of high sehool and had no more than one semester of failure in a eore subjeet in that 
sehool year (English, math, seienee, and soeial studies). 

'’The "total eredits earned toward graduation" measure is the aggregate number of eourse 
eredits earned toward fulfilling the New York State graduation requirements. The eredit 
requirements are as follows: 31 eore subjeet eredits, ineluding 8 eredits eaeh of English and 
soeial studies; 6 eredits eaeh of math and seienee; 2 eredits of arts; 1 eredit of health; and 13 
additional eredits, ineluding 4 eredits of physieal edueation, 2 eredits of a foreign language, and 
7 eredits of eleetives. 
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